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PATTERN SEARCH METHODS FOR LINEARLY CONSTRAINED MINIMIZATION 


ROBERT MICHAEL LEWIS * AND VIRGINIA TORCZON t 


Abstract. Wc extend pattern search methods to linearly constrained minimization. We develop a 
general class of feasible point pattern search algorithms and prove global convergence to a Karush-Kuhn- 
Tucker point. As in the case of unconstrained minimization, pattern search methods for linearly constrained 
problems accomplish this without explicit recourse to the gradient or the directional derivative. Key to the 
analysis of the algorithms is the way in which the local search patterns conform to the geometry of the 
boundary of the feasible region. 

Key words. Pattern search, linearly constrained minimization. 

Subject classification. Applied and Numerical Mathematics 

1. Introduction. This paper continues the line of development in [5, 6, 11] and extends pattern search 
algorithms to optimization problems with linear constraints: 

^ 1 ) minimize f(x) 

subject to £ < Ax < u, 

where / : lR n —> IR, x e IR n , A e Q mXn , £, u € IR m , and £ < u. Wc allow the possibility that some of the 
variables arc unbounded cither above or below by permitting £ i: u z = ±oo, i e {1, • • • , m}. Wc also admit 
equality constraints by allowing £i = u x . 

We can guarantee that if the objective / is continuously differentiable, then a subsequence of the iterates 
produced by a pattern search method for linearly constrained minimization converges to a Karush-Kuhn- 
Tucker point of problem (1.1). As in the case of unconstrained minimization, pattern search methods for 
linearly constrained problems accomplish this without explicit recourse to the gradient or the directional 
derivative. We also do not attempt to estimate Lagrange multipliers. 

As with pattern search methods for bound constrained minimization [5], the pattern of points over which 
we must search in the worst case will, when we are close to the boundary, conform to the geometry of the 
boundary. The general idea, which also applies to unconstrained minimization [6], is that the pattern must 
contain search directions that comprise a set of generators for the cone of feasible directions. Wc must be a 
bit more careful than this; we must also take into account the constraints that are almost binding in order 
to be able to take sufficiently long steps. In the bound constrained case this turns out to be simple to ensure 
(though in §8.3 we will sharpen the results in [5]). In the case of general linear constraints the situation is 
more complicated. 
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Practically, wc imagine pattern search methods being most applicable in the case where there are rela 
tively few linear constraints besides simple bounds on the variables. This is true for the applications that 
motivated our investigation. Our analysis does not assume non-degeneracy, but the class of algorithms we 
propose will be most practical when the problem is nondegenerate. 

2. Background. After we presented this work at the 16th International Symposium on Mathematical 
Programming in Lausanne, Robert Mifflin brought to our attention the work of Jerrold May in [7], which 
extended the derivative- free algorithm for unconstrained minimization in [8] to linearly constrained problems. 
May proves both global convergence and superlinear local convergence for his method. To the best of 
our knowledge, this is the only other provably convergent derivative-free method for linearly constrained 
minimization. 

Both May’s approach and the methods described here use only values of the objective at feasible points 
to conduct their searches. Moreover, the idea of using as search directions the generators of cones that 
arc polar to cones generated by the normals of faces near the current iterate appears already in [7]. As 
our analysis will indicate, this is unavoidable if one wishes to be assured of capturing any possible feasible 
descent in / using only values of / at feasible points (as would be the case in a derivative- free feasible point 
minimization method). 

On the other hand, there are significant differences between May’s work and the approach wc discuss 
here. May’s algorithm is more obviously akin to a finite- difference quasi-Newton method. Most significantly, 
May enforces a sufficient decrease condition; pattern search methods do not. Avoiding a sufficient decrease 
condition is useful in certain situations where the objective is prone to numerical error. The absence of 
a quantitative decrease condition also allows pattern search methods to be used in situations where only 
comparison (ranking) of objective values is possible. 

May also assumes that the binding constraints are never linearly dependent i.e., non-degeneracy. Our 
analysis, which is based on the intrinsic geometry of the feasible region rather than its algebraic description, 
handles degeneracy (though from a practical perspective, degeneracy can make the calculation of the pattern 
expensive). On the other hand, we must place additional algebraic restrictions on the search directions since 
pattern search methods require their iterates to lie on a rational lattice. To do so, we require that the 
matrix of constraints A in (1.1) be rational. This mild restriction is a price paid for not enforcing a sufficient 
decrease condition. 

May’s algorithm also has a more elaborate way of sampling / than the general pattern search algorithm 
wc discuss here. This and the sufficient decrease condition he uses enables May to prove local superlinear 
convergence, which is stronger than the purely global results wc prove here. 

Notation. We denote by R, Q, Z, and N the sets of real, rational, integer, and natural numbers, 
respectively. The ith standard basis vector will be denoted by e*. Unless otherwise noted, norms are 
assumed to be the Euclidean norm. We will denote the gradient of the objective by g(x). 

We will use f2 to denote the feasible region for problem (1.1): 

n^{xelR n | £< Ax <u} . 

Given a convex cone K wc denote its polar cone by K°\ K° is the set of v € lR n such that (t>,uO < 0 
for all w e K, where (v, ie) denotes the Euclidean inner product. 

If Y is a matrix, y e Y means that the vector y is a column of Y . 

3. Pattern Search NIethods. We begin by defining the general pattern search method for the linearly 
constrained problem (1.1). 
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3.1. The Pattern. The pattern for linearly constrained minimization is defined in a way that is only 
slightly less flexible than for patterns in the unconstrained case. In [11], at each iteration the pattern P k is 
specified as the product P k = BC k of two components, a fixed basis matrix B and a generating matrix C k 
that can vary from iteration to iteration. This description of the pattern was introduced in the unconstrained 
case in order to unify the features of such disparate algorithms as the method of Hooke and Jeeves [4] and 
multidirectional search (MDS) [10]. In the case of bound constrained problems [5], we introduced restrictions 
on the pattern itself rather than on B and C k independently, but maintained the artifice of the independence 
of the choice of the basis and generating matrices. 

For linearly constrained problems, we will ignore the basis -i.e., we will take D — I and work directly 
in terms of the pattern P k . We do this because, as with bound constrained problems, we need to place 
restrictions on P k itself and it is simplest just to ignore B. 

A pattern P k is a matrix P k E Z nXpfc . We will specify p k in §3.5; for now we simply note that p k > n 4- 1. 
There is no upper bound on p k . We partition the generating matrix into components 

( 31 ) Pk = [ r fc L k ]. 

We require that T k € Z nxr * belongs to a finite set of matrices T with certain geometrical properties described 
in §3.5, and that L k E Z, nx ( pk Tk ) contains at least one column, a column of zeroes. The inclusion of a column 
of zeroes is simply a formalism to allow for a zero step, i.e., = x k . Again, we will fully specify r k in 

§3.5, but for now we note that n + 1 < r k < p k - 

Given A k E HI, A k > 0, we define a trial step s k to be any vector of the form sj. = A k c l k , where c l k 
denotes a column of P k = [c\ • • • c£ k ]. We call a trial step s{ feasible if ( x k + s[) E ft. At iteration k, a trial 
point is any point of the form x l k = x k T s l k , where x k is the current iterate. 

3.2. The Linearly Constrained Exploratory Moves. Pattern search methods proceed by conduct- 
ing a scries of exploratory moves about the current iterate x k to choose a new iterate x k ^i = x k + s k , for 
some feasible step s k determined during the course of the exploratory moves. The hypotheses on the result 
of the linearly constrained exploratory moves, given in Fig. 3.1, allow a broad choice of exploratory moves 
while ensuring the properties required to prove convergence. In the analysis of pattern search methods, these 
hypotheses assume the role played by sufficient decrease conditions in quasi-Newton methods. Note that the 

L s k E A k P k = A k [r fc L k \. 

2- {x k + s k ) E ft. 

3. If min { f(x k + y) \ ye A k T k and (x k + y) E ft } < /(**), 
then f(x k +5*) < f(x k ). 

Fig. 3.1. Hypotheses on the result of the linearly constrained exploratory moves. 

only change from the unconstrained case is the requirement that the iterates must be feasible. 

3.3. The Generalized Pattern Search Method. Fig. 3.2 states the general pattern search method 
for minimization with linear constraints. To define a particular pattern search method, we must specify the 
pattern P k , the linearly constrained exploratory moves to be used to produce a feasible step s k , and the 
algorithms for updating P k and A k . 

3.4. The Updates. Fig. 3.3 specifies the rules for updating A fc . The aim of the update of A k is to force 
decrease in f(x). An iteration with f(x k + s k ) < f(x k ) is successful ; otherwise, the iteration is unsuccessful. 
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Let xo 6 ft and Ao > 0 be given. 

For k = 0, 1, • • ■ , 

a) Compute f(x k ). 

b) Determine a step s k using a linearly constrained exploratory moves algorithm. 

c) If /(x fc + s k ) < }{x k ), then x k +i = x k + s k . Otherwise x*-fi = x k . 

d) Update P k and A k . 

Fig. 3.2. The general pattern search method for linearly constrained problems. 

As is characteristic of pattern search methods, a step need only yield simple decrease, as opposed to sufficient 
decrease, in order to be acceptable. 

Let r € Q, r > 1, and {u;o> tfi , • • • , w^} C Z, t«o < 0, and w>i > 0, i = 1 Let 0 = t Wo , and 

A fc €A = {r“V-- ) T**}. 

a) If f(x k + s k ) > f(x k ), then A fc +i = 6A k . 

b) If f(x k + s k ) < f(x k ), then A*,+ i = A fc A fc . 

Fig. 3.3. Updating A k . 

The conditions on 0 and A ensure that 0 < 0 < 1 and A t > 1 for all A* € A. Thus, if an iteration is 
successful it may be possible to increase the step length parameter Afc, but A k is not allowed to decrease. 
These conditions are identical to those for the unconstrained case. 

3.5. Geometrical restrictions on the pattern. In the case of linearly constrained minimization the 
pattern P k must reflect the geometry of the feasible region when the iterates are near the boundary. Pattern 
search methods do not approximate the gradient of the objective, but instead rely on a sufficient sampling of 
f(x) to ensure that feasible descent will not be overlooked if the pattern is sufficiently small. We now discuss 
the geometrical restrictions on the pattern that make this possible in the presence of linear constraints. 

3.5.1. The geometry of the nearby boundary. We begin with the relevant features of the boundary 
of the feasible region near an iterate. Let a[ be the ith row of the constraint matrix A in (1.1), and define 

Ae t = { x | a[x = U } 

A Ui = { x | ajx = Ui } . 

These are the boundaries of the half-spaces whose intersection defines Q. Set 

(e) = ( x G f) | dist(x, A^ ) < e } , 

= { x € fi | dist(x, A Ut ) < e } , 


and 

m 

00(e) = U(^,(e)uafl u ,(e)). 

t= 1 

Given xgH and e > 0 we define the index sets 


I({x,e) = { i | x e dn ei (e) } 
I u (x,e) = { i | x e dn u ,(e) }. 


(3.2) 

(3-3) 
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v ti (x, e) = ~a x 


For i £ It{x,£) we define 
(3.4) 

and for i £ I u (x,e) we define 
( 3 - 5 ) v u .(x y £) = a,. 

These are the outward pointing normals to the corresponding faces of ft. 

Given x G ft we will define the cone K{x,e) to be the cone generated by the vectors i/ e .{x,e) for 
i £ Ie(: r y e) and i/ Ui (x y e) for i £ I u (x y £). Recall that a convex cone K is called finitely generated if there 
exists a finite set of vectors {iq, ■ ■ • ,v r } (the generators of K) such that 

K = | V I f = A iVi, A; > 0, i — 1, • • • , r 

L i=i 

Finally, let Pk(x,c) and Pk°( x ,e) be the projections (in the Euclidean norm) onto K(x,e) and K°(x,e), 
respectively. By convention, if K(x,e) = 0, then K°(x y e) = lR n . Observe that AT(x, 0) is the cone of 
normals to ft at x, while K°(x, 0) is the cone of tangents to ft at x. 

The cone K(x, £), illustrated in Fig. 3.4, is the cone generated by the normals to the faces of the boundary 
within distance e of x. Its polar K°{x y £) is important because if £ > 0 is sufficiently small, we can proceed 
from x along all directions in K (x,£) for a distance S > 0, depending only on £, and still remain inside the 
feasible region. This is not the case for directions in the tangent cone of the feasible region at x, since the 
latter cone does not reflect the proximity of the boundary for points close to, but not on, the boundary. 

3.5.2. Specifying the pattern. We now state the geometrical restriction on the pattern P k . We 
require the core pattern T k of P k to include generators for all of the cones K°(x k , e), 0 < e < e*, for some 
£* > 0 that is independent of k . 

In §3.1 we required F k to be one of a finite set of integral matrices T that is independent of k. Thus T 
will contain generators for all of the cones h°(x ky £), 0 < £ < £*. Note that as e varies from 0 to £* there is 
only a finite number of distinct cones K(x k ,e) since there is only a finite number of faces of ft. This means 
that the finite cardinality of T is not an issue. There remains the question of constructing sets of generators 
that are also integral; we address the issue of constructing suitable patterns in §8. However, we will see that 
the construction is computationally tractable, and in many cases is not particularly difficult. 

If x k is “far” from the boundary in the sense that K(x ki e) = 0, then K°(x kl e) = JR n and a set of 
generators for A (xfc,£) is simply a positive basis for lR n [2, 6]. (A positive basis is a set of generators for 
a cone in the case that the cone is a vector space.) Thus, if the iterate is suitably in the interior of ft, the 
algorithm will look like a pattern search algorithm for unconstrained minimization [6], as it ought. On the 
other hand, if x k is near the boundary, A ( x ky £ ) ^ 0 and the pattern must conform to the geometry of the 
boundary, as depicted in Fig. 3.4. 

The design of the pattern reflects the fundamental challenge in the development of constrained pattern 
search methods. We do not have an estimate of the gradient and consequently we have no idea which con- 
straints locally limit feasible improvement in /(x). In a projected gradient method one has the gradient and 
can detect the local interaction of the descent direction with the boundary by conducting a line-search along 
the projected gradient path. In derivative- free methods such as pattern search we must have a sufficiently 
rich set of directions in the pattern since any subset of the nearby faces may be the ones that limit the fea- 
sibility of the steepest descent direction, which is itself unavailable for use in the detection of the important 
nearby constraints. 
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Fig. 3.4. The situation near the boundary. 

4. Convergence analysis. The following hypotheses underlie our analysis: 

Hypothesis 1 . The constraint matrix A is rational 

The rationality of A is a simple way of ensuring that we can find a rational lattice that fits inside the feasible 
region in a suitable way. 

Hypothesis 2. The set Ln(xo) = {xefi | f{x) < f(xo) } is compact. 

Hypothesis 3. The objective f(x) is continuously differentiable on an open neighborhood D of Lh(xq). 
Let Pn be the projection onto Q. For feasible x, let 

q(x) = Pq{x - g(x)) - x. 

Note that because the projection P u is non-expansive, q(x) is continuous on fL The following proposition 
summarizes properties of q that we will need, particularly the fact that x is a constrained stationary point 
for (1.1) if and only if q(x) = 0. The results are classical; see §2 of [3], for instance. 

Proposition 4.1. Let x e H. Then 


II q(x) || < || g(x) || 

and x is a stationary point for problem (1.1) if and only if q(x) — 0. 

We can now state the first convergence result for the general pattern search method for linearly con- 
strained minimization. 

Theorem 4.2. Assume Hypotheses 1-3 hold. Let {x k } be the sequence of iterates produced by the 
generalized pattern search method for linearly constrained minimization (Fig. 3.2). Then 

liminf || q{xk) || = 0 . 


We can strengthen Theorem 4.2 in the same way that we do in the unconstrained and bound constrained 
cases [5, 11], by adding the following hypotheses. 

Hypothesis 4. The columns of the generating matrix Ck remain bounded m norm , i.e there exists 
C > 0 such that for all k 7 C > \\c l k \\, for all i = 1, • ■ • 

Hypothesis 5. The original hypotheses on the result of the linearly constrained exploratory moves are 
replaced with the stronger version given in Fig. 4-1. 

Hypothesis 6 . We have limfc-++oo A k = 0. 
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1. Sk € AfcPfc = A*; [Tfc Lit}. 

2. (x* +s fc ) € n. 

3. If min { /(x; 1 + y) | y € A^Tfe and (x* + y) € Cl } < /(xfc), 

then /(xfc + sit) < min { f(xk + y) | 2 / € A^T/; and (xfc + y) € ft }. 

Fig. 4.1. Strong hypotheses on the result of the linearly constrained exploratory moves. 


Note that we do not require step lengths to be monotone non-increasing. 

Then we obtain the following stronger result. 

Theorem 4.3. Assume Hypotheses 1 6 hold. Then for the sequence of iterates {x/c} produced by the 
generalized pattern search method for linearly constrained minimization (Fig. 3.2), 

lim ||<z(x/fc)|| = 0 . 

k — > + 00 


5. Results from the standard theory. We will need the following results from the analysis of pattern 
search methods in the unconstrained case. For the proofs, see [11]; these results generalize to the linearly 
constrained case without change. Theorem 5.1 is central to the convergence analysis for pattern search 
methods; it allows us to prove convergence for these methods in the absence of any sufficient decrease 
condition. 

Theorem 5.1. Any iterate xn produced by a generalized pattern search method for linearly constrained 
problems (Fig. 3.2) can be expressed in the following form: 

N- 1 

(5.1) x N =xo+{P rL,, a- r "”)A 0 B'frz k , 

k = 0 

where 

• xo is the initial guess, 

• (3/ a = r, with q,/3 € N and relatively prime , and r is as defined in the rules for updating 
(Fig. 3.3), 

• riB and r UD &re integers depending on TV, 

• A 0 is the initial choice for the step length control parameter , 

• B is the basis matrix , and 

• Zk G Z n , A; = 0, * • ■ , TV — 1. 

Recall that in the case of linearly constrained minimization, B = I. 

The quantity A^ regulates step length as indicated by the following. 

Lemma 5.2. (i) There exists a constant > 0, independent of k, such that for any trial step s l k ^ 0 
produced by a generalized pattern search method for linearly constrained problems we have || s\ || > (*A fc . 

(ii) If there exists a constant C > 0 such that for all k , C > ||c^||, for all i = 1, • • • ,Pfc, then there exists 
a constant ifc* > 0, independent of k, such that for any trial step s l k produced by a generalized pattern search 
method for linearly constrained problems we have A k > V 7 *ll s ]cll* 

6. Results concerning the geometry of polyhedra. We need a number of results concerning the 
geometry of polyhedral and convex cones. We begin with a classical result on the structure of finitely 
generated cones. 
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Theorem 6.1. Let C be a finitely generated convex cone in lR n . Then C is the union of finitely 
many finitely generated convex cones each having a linearly independent set of generators chosen from the 
generators of C. 

Proof. See Theorem 4.17 in [12]. 0 

COROLLARY 6.2. Let C be a finitely generated convex cone in IR n with generators {t> l5 ■ • . ,v r }. Then 
there exists Ce .2 > 0, depending only on such that any z € C can be written in the form 

z i Ait^ with A > 0 and j| A j| fC c§ 2 || z [j. 

Proof Theorem 6.1 says that we can write z in the form z — A ij v i;) where r z < r, A* > 0, and 

the matrix V z = [v^ * • • v irx ] has full column rank. The full column rank of V z means that the induced linear 
transformation is one-to-one, so if V+ is the pscudoinverse of V z , then (A^ , • • • , A tr ) T = V z + z. If we define 
A via 


At = J At, if i = ij, 

[ 0 otherwise, 

then A > 0, z = V\ , and || A || < || V+ || || z ||. Since the matrix V z is drawn from a finite set of possibilities 
(e.g., the set of all subsets of {vi, • • « , tv}), we can find the desired constant C6. 2l independent of z. □ 

Let C be a closed convex cone in IR n with vertex at the origin and let C° be its polar. Given any vector 
z, we will denote by z c and z c ° the projections of z onto the cones C and C ° , respectively. The classical 
polar decomposition of z [9, 13] allows us to express z as 

Z = Zc + Z C o , 


where (z c <z C °) ~ 0. 

Proposition 6.3. Suppose the cone C is generated by {n, • -,v r }. Given 7 > 0, there exists ce. 3 > 0, 
depending only on {17, • • • , v r } and 7, such that if || z c || > 7 || 2 ||, zfi=0, then 


max 

\<i<r 


T 

Z Vi 


Vi 


> C6.3- 


Proof By Corollary 6.2, we have C6. 2 > 0, depending only on {v u ■ ■ ■ , v r }, such that we can write z c as 
z c = Z)Li with II A || < cg .2 || z c || and A > 0. Then 

z T z c = ^2 i zTy i< 

i = 1 

so for some i wc must have 

XiZ T Vi > } z T z c = 1 II Zc II 2 • 

Since || A || < C 6.2 || zc ||, we obtain 

rp 1 1 

2 Vi > II zc II - 

r C6.2 

If we let 


v* = max II 

1 <i<r " 
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and apply the hypothesis || zc || > 7 || z ||, we obtain 


r 1 1 1 M 

* > 7 - — — I Vi | 

r ce . 2 v 


and the desired result. □ 

Corollary 6.4. Given 7 > 0, there exists eg . 4 > 0 , depending only on A and 7, for which the following 
hold. For any xGll, £ > 0, 

1- If II ZK(x,e) II > 7 II z II, Z ± 0 , then 


T 

Z Vi 


max 

l<i<r 


> Q).4 • 


where {ui, • ■ ■ , u r } are the generators of K(x,e) defined in (3.4) (3.5). 
2- tf || ZK*{x t e) || > 7 \\z ||, ^ 7 ^ 0, then 


T 

z Vi 


max 

Kt<r 


> C6.4* 


where {iq, • ■ , u r } are the generators of K°{x,e) required in § 3.5.2 to be in the set T . 

Proof The corollary follows from the observation that since K(x,e) is generated by subsets of the rows 
of A, K(x,e ) can be one of only a finite number of possible cones. Consequently K°(x,e) will also be one 
of only a finite number of possible cones. Applying Proposition 6.3 to each of these cones in turn (with the 
generators (3.4) (3.5) for K(x,e) and the generators in T for A'°(x,£)) and taking the minimum yields the 
corollary. □ 

Let 


«* = max {|| Oi ||} 

l<i<m 

a. = min {|| a t ||>. 

l<i<m 

We may assume, without loss of generality, that a* > 0 . The next proposition says that if x E Q is close to 
the boundary of Q and a sufficiently long step in the direction w remains feasible, then w cannot be “too 
normal” to d£l near x. 

Proposition 6.5. Given rj > 0 , there exist R > 0 and cq , 5 > 0 7 depending only on A, such that if 
0 < £ < R, x e d£l(e), || w || > 7], and (x + w) € then || Pk o { x ,c)W || > Ce>.$ || w ||. 

Proof A simple calculation shows that the distance from any point x to the affine subspacc defined by 
aj z = b is | b — ajx | / || a* ||. Thus, if the distance from x to aj z = b is no more than e, then 

(6.1) 6 - £ || ai || < afx < b + £ || ai || . 

Also note that the distance from the affine subspacc aj x = £i to aj x = Ui is 

Ui — L 

kT* * 

Now let C 6.4 > 0 be the constant from Corollary 6.4. Set 

R _ c 6-4 a * V 
~ a* 2 

and consider any £ such that 0 < e < R. 
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By our convention, if K{x,e) = 0, then K°(x,e) — lR n and the proposition holds with C6. 5 = 1 since 
the projection onto K°(x, e) is the identity. Thus we need only consider the case where K(x,e) ^ 0- From 

(6.1) , if x € ^en for i € Ie(x,e) U I u (x,e) we have either 

ii < afx < £i 4- e || a, || 
or 

Ui — e || a,i || < afx < Ui. 

This gives us two cases to consider: 

1. Suppose £i < ajx < £{ + e \\ a,i \\. Since (x 4- w) e ft, we have 

U < afx + af wk 4- af wk° , 
where w = wk 4- Wk° is the polar decomposition of w , whence 

0 < ajx -£i + qJwk + af w K o < e || a % || + af w K + a* || w K ° || - 
Since in this case ve x (x, e:) = — ai (sec (3.4)), we have 

(6.2) vf.wK < e || || + a* || w K o || < a*(e + || w K * ||). 

2. If, on the other hand, — e || a t || < afx < Uj, then, since (x + w) G f 2, 

afx + af wk 4- afwK° < a* 

yields 

0 < Uj - a- x - af w K - af w K ° < £ || a* || - af w K + a* || w K o || . 

In this case v Ux ( x i e) = a* (see (3.5)), so 

(6.3) Vu, w K < «|| «» II + II II < a*(e+ || ||). 

Now consider the generators ve { , u Ui for K(x,e). If we apply Corollary 6.4 to K(x, e) and z = wk (with 
7 = 1), then for one of the generators, which we will simply call v, we have v t wk > c$a || w K || || v ||. From 

(6.2) and (6.3) we obtain 

06.4 || v || || wk || < v t w k < a*(e 4- || w K o ||) 


or 

II WK II < -^-(e+|| WK« II). 
C6.4G* 

Since || w || < || xvk || + || tOK° ||, we have 


w || < 


^6.4 a* 

By our choice of t and the hypothesis j w || > r/. we obtain 


(e+ || w/c» ||) + || wk ° 


w I! < 


f + (i + -^-) II «*. 

2 \ C6. 4 a* J 

(i + — ) ii 

\ 05.4a* J 


< + 


Wk° 
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and finally 


II w || < 2 [ 1 + 


— ) 


W K o , 

\ ^6,4^ / 

giving us the desired bound. □ 

The next proposition relates the global geometry of fi and the local geometry of fi near a feasible point 
x. It is an elaboration of the observation that a convex set lies on one side of a hyperplane tangent to its 

boundary. The result is also true without the restriction of x to a compact subset of fi, but this assumption 
shortens the proof. 

POSITION 6.6. Given 7 > 0, r; > 0, and a compact set S C fi, there exist R > 0 and c$ 6 > 0 such 
that forallxeS and alle,0<e<R, if || w || < 7 and || P n (x + w) - x || > n , then 

II Pk°(x,c)W || > C6.6 || Pj(o(x,e) (Pq(x + w) - x) || . 


Proof. Suppose the proposition is not true. Then, for all ,, trying c,., = R= l/j, wc can find x, € 5 
£j WlUCh ° ~ and w i for which II II < 7 and || P n ( Xj + Wj ) - Xj || > r,, such that 

II PK°(x } ,e j )Wj || < J || Pro ( X] ,e 3 ) (Pfl(Xj + Wj) - Xj) || . 

For convenience, let Zj — Pq (x } + Wj) — Xj\ then || Zj || < || Wj || (sec Proposition 4.1). 

Applying the compactness of S and the boundedness of the sequences {«;,■}, { Zj \, we may find a sul> 
sequence for which x jk -> *. € S, w jk - w „ and where || w, || < 7, || || > r/ . and || || < r 

otc z * Pq(x* w*) — by the continuity of the projection Pq, 

Furthermore, since there are only a finite number of possible cones K(x jk ,e jk ), there exists a cone K, 

Z C t t a / t K * ~ K ( Xjk,£jk ' 1 mfinltel y often. By selecting this further subsequence, wc may assume that 
K t = K(x jk ,e jk ) for all k. 

Next we will show that K, C K(x. t 0). By construction, K. is generated by a subset of {±aj i € 
{1, • • • ,m}. If — a; £ K* = K{xj k ,ej k ), then ' 


dist(xj t , Ai x ) < £jk < 1 fj k 


for all k , so 


dist(x*, A ei ) 


— ajx* 


< 


~ Qj X 3k 

II at II 


+ 


® i Q'i 

Kli 


: dist(x Jfc , At x ) + 


Taking the limit as A; —> 00 we sec that dist(x. , A ti ) = 0, so -a, € K{x , , 0). A similar argument, substituting 

n t for £ it shows that if a, € K„ then a, e K(x„ 0). Since K(x„ 0) contains the generators of K„ it follows 
that K* C /T(x*,0), as desired. 

Consequently, K° D K°(x„0) and, since K° = K°(x jk ,e jk ), wc have 

|| P K°(x.,0)W* || < || Pr°W » || = | PK°(x jk ,e jk )W* 
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from which we conclude that 


< 


P K°( x ik .e )k )( w * w jk) 


+ 


P K°(x jk ,e jk ) w }k 


< II VJ+ — w 


3k 




P K°(x ik ,e ik ) z jk hk 


< II W, - W jk II + II z jk II /jk 

< || tn. - w jk II + 7 /jk, 


II P K°(x.,0 )U>* II = 0 ' 

This means that in, is normal to H at x». However, it would follow that z, = 0, a contradiction. □ 

As wc noted at the introduction of K“(x,e), we can proceed from r along all directions in K°(x,c) for 
a distance i > 0, depending only on t, and still remain inside the feasible region. The following proposition 
is the formal statement of this observation. Define 


mm 

1 < i < m 


Ui - 


(6.4) 

This is the minimum distance between the faces of D associated with the constraints that are not equality 
constraints. 

Proposition 6.7. Given e > 0, e < h/2, therv exists 6 6 . 7 > 0 such that for any * G il,ifweK (x,e) 

and || w || < (56.7) then (x + u?) € 

Proof. Let 


$ — (56.7 


£ 

2 


and consider any index t 6 {1, • • -,m}. Wc will show that x + w is feasible with respect to the ith constraint. 
If x £ dfl ti (<r) U dn u ,(e), then U + £ || o* || < ajx < m - e || a ( ||, so 


n 7 t 4- nT W > ti 4- £ I 


>> 0 . _L ( c — 


and 

ajx + ajw < Ui -e|| o, || + || a, || || in || < m - (e - 6) || a, || < u 4 . 

On the other hand, suppose * € 0Q f Je) U dV u ,(e). There arc three cases to consider. First suppose 
x e fb,(e) and x € an u ,(e). Since e < h/2 , this means that £ t = u t (i.e., the constraint is an equality 
constraint). Then, if w G K°(x,e), wc have both (in , -aj < 0 and (in , a { ) < 0, so (in , a l ) = 0. Thus 

l { — ajx + ajw = Ui . 

Next suppose x € «l*(e) but x $ dQ u ,(e). If in G K°(x,e), wc have < 0. Applying (6.1) we obtain 

ti < ajx + ajw<£ t + e || a, || + || a, || || w || < U + II «i II (e + 6) < u t . 

Finally, if x € dSl Ul (e) but x 0 3n <4 (e), then, if in € K°(x,e), (a % ,w) < 0, so 

Ri > ajx + ajw> Ui-e || a t || - || a, || || in || > u t - || «, || (e + 6) > i t . 

Thus (x + tn) satisfies the constraints for all i G {1, • • • ,m), so (x + tn) G fl. □ 
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7. Proof of Theorems 4.2 and 4.3. Given an iterate x k , let g k = g(x k ) and q k = P n (x k - g k ) - x k . 

Let B(x,S) be the ball with center x and radius 8, and let ui denote the following modulus of continuity of 
g: given x £ L n (x 0 ) and e > 0, 

uj(x, e) = sup { 8 > 0 | B(x, 5) C D and || g(y) — g[x ) || < e for all y £ B{x, 8) } . 

Then we have this elementary proposition concerning descent directions, whose proof we omit (see [5]). 

Proposition 7.1. Let s £ lR n and x € Ln(:ro). Assume that g(x) ^ 0 and g{x) T s < -e || s ||. Then, 
*/ll s || <u>{x, §), 

f(x + s)~ f(x) < || s || . 

The next result is the crux of the convergence analysis. All the results of §6 arc brought to bear to show 
that if we are not at a constrained stationary point, then the pattern always contains a descent direction 
along which we remain feasible for a sufficiently long distance. 

Let r be the maximum norm of any column of the matrices in the set T, where T is as in §3.1 and §3.5. 
If A < 8/ T*, then || sj. || < 8 for all sj^ £ I\. 

Proposition 7.2. Let g > 0. Then there exists c 7 . 2 > 0 and 8 7 . 2 > 0 such that if || q k || > g and 
A fc < 8 7 . 2 then there is a trial step s{ defined by a column of A k T k for which (x k + s' ) e fl and 


~9h s k ^ c 7.2 II qk 


4 


Proof. By hypothesis, || g k || is bounded on L u (x 0 y, let g* be an upper bound for || g k ||. Consider e > 0 
sufficiently small that 

1- £ < h/ 2), where e* was introduced in §3.5.2 and h is given by (6.4), and 

2. both Proposition 6.5 and Proposition 6.6 hold, 
where we invoke Proposition 6.6 with 7 = g* since we intend to apply it to w = —g k . 

If x k 6 dil(e), then, since (x k + q k ) £ LI and || q k || > rj. by Proposition 6.5 we have 

C 7 ' 1 ) II P K°(x k ,e)qk || > Ce.5 || q k || . 

This bound also holds if x £ dLl{e), since then K(x,e) = 0 and K°(x,e) = lR n , and necessarily r^, 5 < 1. 
Meanwhile, Proposition 6.6 says that 

II P K°(x k ,e){ 9k) |j ^ ^-6.6 || P K°(x k ,e)9k || , 

so from (7.1) we obtain 

( 7 ' 2 ) II P K°{x k ,e){~9k) || > C 6 . 5 C 6.6 || qk || 

( 7-3 ) > C6.5C6.67?|| 9k ||/5*. 

We require the core pattern of P k to include generators for all of the cones K°(x k ,e), e < e*. 
Therefore some subset of the core pattern steps s', forms a set of generators for K°(x k , e). The lower bound 

(7.3) allows us to apply Corollary 6.4 with 2 = -g k and K°(x k ,e). If we do so and apply (7.2), we sec that 
for some sj. £ A k F k , 

( ~9k , 4) > C6 4 || - 9k || || 4 || 

> C6.4 || P K°(x k ,e)(. 9k) || || 4 || 

> C 6 . 4 C 6 . 5 C 6 6 || Qk || || 4 || 
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Thus we arc assured of a descent direction inside the pattern. 

Now we must show that we can take a sufficiently long step in the direction of this descent direction and 
remain feasible. Proposition 6.7 allows us to do this; given e > 0 we can find 6 such that once A < S/T*, 
then, since sj. 6 K°(x k ,e) and || s{ || < S, we have (x k + s l fc ) G fi. □ 

We now show that if we are not at a constrained stationary point, we can always find a step in the 
pattern is both feasible and yields improvement in the objective. 

Proposition 7.3. Given any rj > 0, there exists S > 0, independent of k , such that if A k < <5 and 
|| q k || > r), the pattern search method for linearly constrained minimization mil find an acceptable step s k ; 
he., f(x k + Sfc) < f{x k ) and ( x k + s k ) € fi. 

If ; i n addition, the columns of the generating matrix remain bounded in norm and we enforce the strong 
hypotheses on the results of the linearly constrained exploratory moves (Hypotheses 4 and 5), then, given any 
r) > 0, there exist 67.3 > 0 and o > 0, independent of k, such that if A k < 67.3 and || q k || > i), then 

f(x k + 1 ) < f(x k ) - a || q k || || s k || . 


Proof. Since g{x) is uniformly continuous on Lq(xq) and Ln(x 0 ) is a compact subset of the open set D 
on which f{x) is continuously differentiable, there exists w, > 0 such that 

for all k for which || q k || > g. Also define 


S = <$ 7.3 = min ( 87 . 2 , u*/T*) . 

Now suppose || q k || > 7 ] and A fc < <5. Since A k < 67 . 2 , Proposition 7.2 assures us of the existence of a 
step s' k defined by a column of A k T k such that [x k + s k ) € it and 


g k s k < -C7.2 II Qk II || | • 


At the same time, we also have 

|| 4 || < A k r* < u), < to (x k , ^ II ll) • 

Hence, by Proposition 7.1, 

f(x k + 4) - f( x k) < II qk II II s k II ' 

Thus, when A k < 6 , f(x k + 4) < f(x k ) for at least one feasible s{ G A k T k . The hypotheses on linearly 
constrained exploratory moves guarantee that if 

min{/(xfc + y) | y € AfcTfc, (x* + y) 6 } < /(^fc), 

then f(x k + s k ) < f(x k ) and (x k -f s k ) € This proves the first part of the proposition. 

If, in addition, we enforce the strong hypotheses on the result of the linearly constrained exploratory 
moves, then we actually have 

/(Xfc + l) - f( x k) < II II II S k II ■ 
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Part (i) of Lemma 5.2 then ensures that 


/(*fc+i) < /(a*) - ^C.A* || q k || • 

Applying part (ii) of Lemma 5.2, we arrive at 

/(**+ 1) < f(Xk) - II q k II II S* || . 

This yields the second part of the proposition with <r = (c 7 . 2 /2 K.V'.- □ 

Corollary 7.4. If limmf*_ +0O || q k || ^ 0, then there exists a constant A, > 0 such that for all k 
A* > A.. 

Proof. By hypothesis, there exists TV and rj > 0 such that for all k > N, || q k || > i q. By Proposition 7.3, 
we can find 6 such that if k > N and A k < 6, then we will find an acceptable step. In view of the update of A k 
given in Fig. 3.3, we are assured that for all k > N, A k > 96. We may then take A* = min{A 0 , • • • , A N ,86}. 

The next theorem combines the strict algebraic structure of the iterates with the simple decrease condi- 
tion of the generalized pattern search algorithm for linearly constrained problems, along with the rules for 
updating A k , to tell us the limiting behavior of A k . 

Theorem 7.5. Under Hypotheses 1 3, liminf fc _^ +00 A k = 0. 

Proof. The proof is like that of Theorem 3.3 in [11]. Suppose 0 < A LB < A k for all k. Using the rules 
for updating A*, found in Fig. 3.3, it is possible to write A k as A k = r Tk A 0 , where r k e Z. 

The hypothesis that A LB < A k for all k means that the sequence {r Tk } is bounded away from zero. 
Meanwhile, we also know that the sequence {A k } is bounded above because all the iterates x k must lie 
inside the set L n (x 0 ) = { x e ft | f(x) < f(x 0 ) } and the latter set is compact; part (i) Lemma 5.2 then 
guarantees an upper bound A UB for {A*}. This, in turn, means that the sequence {r Tk } is bounded above. 

Consequently, the sequence {r r *} is a finite set. Equivalently, the sequence {r*} is bounded above and 
below. 

Next we recall the exact identity of the quantities r LB and r UB in Theorem 5.1; the details are found in 
the proof of Theorem 3.3 in [11], In the context of Theorem 5.1, 

Tlb = r UB= max {r>}. 

0 <k<N 0 <k<N 

If, in the matter at hand, we let 

rm = min {r^} r UB = max {r*,}, 

0<fc<+oo l 0<Jt<+oc l f ’ 

then (5.1) holds for the bounds given in (7.4), and we see that for all k, x k lies in the translated integer 
lattice G generated by x 0 and the columns of 0 rLO a~ r ' JB A 0 I. 

The intersection of the compact set L n (x 0 ) with the lattice G is finite. Thus, there must exist at least 
one point x* in the lattice for which = x+ for infinitely many k. 

We now appeal to the simple decrease condition in part (c) of Fig. 3.2, which guarantees that an iterate 
cannot be revisited infinitely many times since we accept a new step s k if and only if f{x k ) > f(x k f ,s k ) and 
(x k + s k ) e SI. Thus there exists an N such that for all k>N,x k = x„, which implies f(x k ) = f(x k + s k ). 

We now appeal to the algorithm for updating A* (part (a) in Fig. 3.3) to see that A k - 0, thus leading 
to a contradiction. 0 


15 



7.1. The Proof of Theorem 4.2. The proof is like that of Theorem 3.5 in [11]. Suppose that 
lim inffc_+c>o || q k || ^ 0. Then Corollary 7.4 tells us that there exists A, > 0 such that for all k, A k > A,. 
But this contradicts Theorem 7.5. 

7.2. The Proof of Theorem 4.3. The proof, also by contradiction, follows that of Theorem 3.7 in 
[11]. Suppose limsup fc _ +00 || q k || ^ 0. Let e > 0 be such that there exists a subsequence || q{x mi ) || > e. 

Since 

lim inf || q k || = 0, 

fc— ► + OO 

given any 0 Tj £, there exists an associated subsequence li such that 

[| q k || > r) for mi < k < U, II II < V- 

Since Ait — > 0, we can appeal to Proposition 7.3 to obtain for rn, < k < li, i sufficiently large, 

f{x k ) ~ f{x k + 1) > ^ II II II Sk II > ar l II Sk II » 

where a > 0. Summation then yields 

f{x m ,) - f(xii) ^ ar > II Sk II - C> II Xm * ~ Xli II ' 

Since / is bounded below, f(x mt ) - /(x,,) — 0 as i — +oo, so || x TOi - x u || — 0 as i — +oo. Then, because 
q is uniformly continuous, || g(i m .) — qi'Ji , ) || < V, for i sufficiently large. However, 

(7.5) |j q{xm , ) || < || q(x mi ) ~q{xii) II + II 9( x h) II ^ 2r i- 

Since (7.5) must hold for any r?, 0 < tj < e, we have a contradiction (c.g., try t] = §). 

8. Constructing patterns for problems with linear constraints. In this section we outline prac- 
tical implementations of pattern search methods for linearly constrained minimization. The details will be 
the subject of future work. In the process wc also show that under the assumption that A is rational, one can 
actually construct patterns with both the algebraic properties required in §3.1 and the geometric properties 
required in §3.5. 

8.1. Remarks on the general case. We begin by showing that in general it is possible to find rational 
generators for the cones K°(x,e). By clearing denominators we then obtain the integral vectors for T as 
required in §3.1. The construction is an elaboration of the proof that polyhedral cones arc finitely generated 
(sec [12], for instance). The proof outlines an algorithm for the construction of generators of cones. Given 
a cone K we will use V to denote a matrix whose columns are generators of K: 

K={x | x = PA, A > 0 } . 

PROPOSITION 8.1. Suppose K is a cone with rational generators V. Then there exists a set of rational 
generators for K ° . 

Proof. Suppose w € K°\ then (to, v) < 0 for all v € K. Let v = PA, A > 0. Then 

(tO , t>) = (PiV(yT)tO + Ptf(yT)±W , PA) < 0, 

where PaT(p t ) aIR l P (A /(v T )) ± are t *' ic projections onto the nullspacc M(V T ) of P T and its orthogonal 
complement Af{V T ) ± , respectively. Since Af(V T ) J - is the same as the range TZ(V) of P, wc have 

(to , v) = (Piz(V)W i PA) < 0. 
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Let N and R be rational bases for Af(V T ) and R(V) 
via reduction to row echelon form since V is rational. 


respectively; these can be constructed, for instance, 


Let {p\,---,Pt} be a rational positive basis for ff(V T '\ S.mh Q „„ a- ,. 
follows. If N is n x r th^n ifrr- *• , . . ' h positlvc basis can be constructed as 

18 a ra lonal positive basis (with t elements) for lR r (c.g., II = f/ - /]) then 
Nu a rational positive basis for 1 

Meanwhile, if R is a rational basis for R(V), then for some z we have 


whence 


Rn( v)W = Rz, 


, V 


■ j l) r and consider 


Since X t V t Rz < 0 for all A > 0, it follows that V T Rz < 0. Let e = (1, 

C ={* I V t Rz< 0, e T V T Rz > — 1 } . 

Since C is convex and compact, it is the convex hull of its extreme points • r 1 FWri. 
the extreme nninto r,f n *n j c „ pomes fcj, , C s j. f urthermore, note that 

p ts of C will define a set of generators for the cone { z I V T Rz < 0 I r ru„ f 

7 ,s rat ' onai; th ° c “ remc p ° iMs -"J— izrz 

rational cocffi lents. These extreme points, which are the vertices of the polyhedron C, can bo eomputed bv 
any number of vertex enumeration tcchnipuc, (e. g „ see ,1] and the reference. cited thmein). " 

(pi ■ • • V c 8 *° c r M ’ WC S °h tlla ' ” Ca ” c>lprcss as * positive linear combination of the vectors 

1P1 , ,Pt,c u ...,c s }. Moreover, by construction the latter vectors arc rational. □ 

8.2. The nondegenerate ease. As we have seen, the construction of sets of generators for cones 

degeneracy 13 tZ abide d “ °‘ VOTticcs ° f Polyhcdra. However, in the case of non- 

g y absence of any point on the boundary at which the set of binding constraints is linearly 

using thd OB "IT T“ tC T r0<1 “ irCd 8C “ rat0rS “ “ stra ’8irtforward way. This ease is handled in |7| by 

use the LU d Z 77 d “ arch B “ aUSC ”»■*« search directions, we 

ration, aI::T °” " “““ *”• *» b ° ^ ^ « b. done 

set oftI°"7‘° 8 PrOPOSiti °" Sh °"' 5 that ° ncc wc haTO 'de-tiUcd a cone K( Xt , 6) with a linearly independent 
set of generators, wc can construct generators for all the cones K(x k e) 0< £ < 8 

V ITZT'Z 8 2 , SUPT °’ e ,CTSOmel ' K{t ' S) *“ * Unear ‘y set o/ rational generator, 

V Let Nbea ^atonal positive basis for the nullspace of V T . 

Then for any £, 0 < e < 8, a set of rational generators for K°(x e) can he fnnnrt 
N, V{V T V)~ 1 , and ~V{V T V)~ l 7 ’ 6 d 0719 the colu ™ns of 

Proof. Suppose w g K°; then (w,v) < 0 for all v g K Let v - VX A > n < 3 - ,/ , f „ , 

rank, wc have — VX, A > 0. Since V has full column 

K v) = ((I - V(V T V)-'V T ) W + V(V T V)-'V T W, vx) < o 

or (W-«,A) < 0 for ail A > 0 Let { = thcn wc w ({ _ A) < 0 A £ ^ £ Q 

ine matrix N is a positive basis for the range of I - V(V T V\~ l \/T • i, , . 

same as the nuiispaee of I-. Then any . e ^ C L he wriZn^e fin laUe ' S “ bSP “ C ^ ** 

w = N$- F(V' t K)- 1 ^ 
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where < > 0 and { > 0. Thus the columns of N and -V^V)~' are a set of generators for if 

Moreover for e < 4 we obtain K = K(x,t) by dropping generators from V. Without loss of generah y 
we will assume that we drop the first r columns of V, where V has p columns. Then cons.der w € _ 
Proceeding as before, we obtain [V^w.X) < 0 for all A > 0, A„ ■ • - , Ar = 0. If we once again 
J = V T W, then we see that See., -' < 0, while {„ • ■ ■ ,«r are unrestricted m sign. Hence we o am 
of generators fo, R° from the columns of N, the first r columns of ViV^ and their negatives, and the 

Lp«itio78.2 01 leal"! The fbLng construction of patted for linearly 
Under the assumption of non-degeneracy, we know there extern s' such that if 0 < s < . *e < ’ J 

has a linearly independent set of generators. If we knew this s', it would be a convenient choice for 
required in §3.5. The following algorithm implicitly estimates s': it conducts what amounts to a s.fcguar 
backtracking on s at each iteration find a value of eg for which If (»,..) has a linearly independent set 

of generators. 

Given e* independent of k, choose £ k >£»- Then 

1. Define the cone K(x k ,e k ) as in §3.5. , . 

2 Let V represent the matrix whose columns are the generators ^ (**=,**) and ^ (**, e k ) o ( fc, *) 
(defined in (3.4) (3.5)). Determine whether or no, V has full column rank. If so, go to Step 

Otherwise, reduce £ fc just until | I t {x k , e k )\ + | /„(**, e k )\ is decreased. Return to Step 1. 

.... /v fnr the ranee of I -V(V T V)~ 1 V T . This can be done via reduc- 

3. Construct a rational positive basis N for the range ot t ) i; , l/T m-i t/r \ 

tion to row echelon form, or simply by taking the columns of the matrices ± ( ( 

4. Form the matrix F*, = \N V(V T V) 1 -V(V V) ]• , , . ,. ... 

Unde, the assumption of no., -degeneracy, Sr will remain bounded away from 0 as a function of k. implicitly 

giving us the £* introduced in §3.5.2. 

8 3 The case of bound constraints. Matters simplify enormously in the ease of bound constraints, 
previously considered in [5], We will briefly discuss the specialisation to bound constrained minimisation 

and in the process sharpen the results in [5]. 

In the case of bound constraints we have 

minimize / (z) 

(^•l) subject to l < x < u. 

Again, we allow the possibility that some of the variables are unbounded either above or below by permitting 

^'In^tll^c^of bound 'constraints wc know o prior, the possible generators of the cones if (x,e) and 
if- (x s) For any x £ !1 and any e > 0 the cone K(x,s) is generated by some subset of the coordinate 

vceL ±e. If K(x,s) is generated by where <E » *—<* * 

the set —v ... -« together with a positive basis fo, the orthogonal complement of the space spanned y 

„. . . . This orthogonal complement simply corresponds to the remaining coordinate directions. 

This simplicity allows us to prescribe in advance patterns that work for all *(x,e). In 5| we gave 
the prescription r. - |i - /). This choice, independent of k. includes generators for all possible if (*,e). 

all the variables are bounded, then one can make a choice of Ft that is independent of 
but mom parsimonious in the number of directions. Let x,„ - ■ ■ , x„ be the variable, with either a owe, o 

upper bound- then Ts should include the coordinate vectors iei,,-- .ie., together wit aposiive asis 

the orthogonal complement of the linear span of e,„ • - , e,. ; a positive basis for the orthogonal comp emcn 
can have as few as (n - r) + 1 elements. 
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The choice of T* — [I J] in [5] requires, in the worst case, 2 n objective evaluations per iteration. The 
more detailed analysis given here leads to a reduction in this cost if not all the variables are bounded. If only 
r < n variables are bounded, then we can find an acceptable pattern containing as few as 2r + ((n — r) + 1) = 
n + r + 1 points. 

Finally, note that if general linear constraints arc present but A has full row rank (i.e., there are no more 
than n constraints and they are all linearly independent), then one can carry out a construction similar to 
that for bound constraints. 

9. Conclusions. We have introduced pattern search algorithms for solving problems with general lin- 
ear constraints. Wc have shown that under mild assumptions we can guarantee global convergence of pattern 
search methods for linearly constrained problems to a Karush-Kuhn-Tucker point. As in the case of uncon- 
strained minimization, pattern search methods for linearly constrained problems accomplish this without 
explicit recourse to the gradient or the directional derivative. In addition, wc have outlined particular in- 
stances of such algorithms and shown how the general approach can be greatly simplified when the only 
constraints arc bounds on the variables. The effectiveness of these techniques will be the subject of future 
work. 
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